In recent times, there has been a dramatic shift in consumer photography. Once dominated by dedicated cameras and photography equipment, a majority of pictures today are taken via camera-equipped mobile devices such as smartphones. Although smartphones offer a great deal of convenience and portability, the quality of the resulting photographs is often quite poor. The discrepancy in image quality between smartphones and dedicated cameras stems from the small form factor of smartphones, which limits the available optics and image sensors. One way to enhance the quality of images produced by a smartphone is via computational image processing techniques such as high dynamic range (HDR) imaging, multi-frame noise reduction, synthetic aperture, flash-no-flash photography, super resolution, and video de-shake. While the above-mentioned computational image processing techniques offer vast improvements in the quality of images produced by smartphones, implementing these techniques in hardware is often expensive in terms of power consumption, area consumption, and/or cost.
Generally, current smartphones apply one of two paradigms in their implementation of image signal processing hardware. A first paradigm uses a general-purpose processor (e.g., the main processor of the smartphone) to implement one or more of the computational image processing techniques discussed above. Although these general-purpose processors are capable of performing a wide range of computational image processing techniques, they are highly inefficient at doing so. Specifically, the overhead associated with predicting, fetching, decoding, scheduling, and committing instructions in a general-purpose processor adds significant computation time to the image processing techniques, especially when considering the particular data storage structures, data flow, and data locality of computational image processing techniques in general.
A second paradigm uses separate application specific image signal processing hardware (i.e., accelerators) for each one of the computational image processing techniques implemented by the smartphone. Although each one of the application specific image processors may provide an improvement in computational efficiency over a general-purpose processor as large as three orders of magnitude, separate hardware for each one of the computational image processing techniques consumes a large amount of area in the smartphone, and further adds significant expense to the device. Further, as new computational image signal processing techniques are developed, dedicated hardware for implementing the new technique must similarly be developed, which prevents implementation of the new technique in currently deployed devices.
Accordingly, there is a need for compact image signal processing hardware that is capable of efficiently performing a variety of computational image processing techniques.